Monoidify! Monoids as a Design Principle for Efficient MapReduce Algorithms

The purpose of this short paper is to share a recent observation I made in the context of my introductory graduate course on MapReduce at the University of Maryland. It is well known that since the sort/shuffle stage in MapReduce is costly, local aggregation is one important principle to designing efficient algorithms. This typically involves using combiners or the so-called in-mapper combiner technique [5]. However, can we be more precise in formulating this design principle for pedagogical purposes? Simply saying“use combiners”or“use in-mapper combining” is unsatisfying because it leaves open the obvious question of how? What follows is my attempt to formulate a more precise design principle in terms of monoids—the idea is quite simple, but I haven’t seen anyone else make this observation before in the context of MapReduce. Let me illustrate with a running example I often use to illustrate MapReduce algorithm design, which is detailed in Lin and Dyer [5]. Given a large number of key–value pairs where the keys are strings and the values are integers, we wish to find the average of all the values by key. In SQL, this is accomplished with a simple group-by and Avg. Here is the näıve MapReduce algorithm: